Fix driver crash on Spark 4.2 (Jetty 12 EE10 relocation) - #82

Problem

DataFlint does not work on Apache Spark 4.2 — and it fails hard: the exception
propagates out of SparkDataflintDriverPlugin.registerMetrics, so
SparkContext initialization fails and the entire Spark application dies at
startup. It is not a degraded-UI failure.

java.lang.ClassNotFoundException: org.eclipse.jetty.servlet.ServletContextHandler
  at org.apache.spark.dataflint.api.DataflintJettyUtils$.getClassForName$1(DataflintJettyUtils.scala:26)
  at org.apache.spark.dataflint.api.DataflintJettyUtils$.createStaticHandler(DataflintJettyUtils.scala:30)
  at org.apache.spark.dataflint.api.Spark4PageFactory.addStaticHandler(Spark4PageFactory.scala:45)
  at org.apache.spark.dataflint.DataflintSparkUICommonInstaller.loadUI(DataflintSparkUICommonLoader.scala:119)
  at io.dataflint.spark.SparkDataflintDriverPlugin.registerMetrics(SparkDataflintPlugin.scala:27)

Spark 4.2 upgraded to Jetty 12, which relocated the shaded servlet classes into
the EE10 package:

DataflintJettyUtils.getClassForName probes only
org.sparkproject.jetty.servlet.* and then org.eclipse.jetty.servlet.*, so on
4.2 both lookups miss.

Affects both published artifacts — io.dataflint:spark_2.13:0.9.9 and
io.dataflint:dataflint-spark4_2.13:0.9.9.

Fix

Probe the EE10 packages first and fall back to the older names, so one build
keeps working across 3.x, 4.0, 4.1 and 4.2.

val candidates = Seq(
  s"org.sparkproject.jetty.ee10.servlet.$className",
  s"org.sparkproject.jetty.servlet.$className",
  s"org.eclipse.jetty.ee10.servlet.$className",
  s"org.eclipse.jetty.servlet.$className"
)

Everything downstream is reflective (setInitParameter, setContextPath,
addServlet) and all of those methods exist on the EE10
ServletContextHandler, so no other change was needed. On failure the error now
lists every package tried instead of surfacing the last ClassNotFoundException.

Also adds example_4_2_0, mirroring example_4_0_1. It doubles as a regression
test: without this change it cannot start a SparkSession at all.

Verification

Apache Spark 4.2.0, local mode, Java 17 (Corretto), Scala 2.13:

• ✅ driver starts and the plugin registers
• ✅ DataFlint tab appears in the Spark UI nav bar
• ✅ GET /dataflint/ → 200
• ✅ GET /dataflint/applicationinfo/json/ → 200 with real data:

  { "info": { "id": "local-1787212371032", "name": "Spark 4.2 - What's New",
              "attempts": [ { "appSparkVersion": "4.2.0", ... } ] },
    "environmentInfo": { "driverXmxBytes": 8589934592 } }

• ✅ sqlmetrics / cachedstorage JSON endpoints respond
• ✅ sbt example_4_2_0/compile succeeds
• ✅ no regression expected on 3.x/4.0/4.1 — the old names are still probed, just
  after the EE10 ones

Exercised against a real workload (~90 SQL executions: Arrow UDFs, multi
COUNT(DISTINCT), Datasketches, nearestByJoin, Delta MERGE).

Known follow-up — NOT fixed here

With the backend healthy on 4.2, the web UI still renders only a loading
spinner:

• #root contains only a MuiCircularProgress
• zero XHR requests are issued from the page
• no JS console errors
• yet fetch('/dataflint/applicationinfo/json/') from that same page returns 200
  with data

Reproduced with the published 0.9.9 static assets (hybrid jar: official 0.9.9
contents with only DataflintJettyUtils swapped), so it is independent of this
change. Since the app never issues a first request it looks like a bootstrap /
runtime-config step rather than an API incompatibility.

Two smaller observations while debugging:

• SPA sub-routes (/dataflint/status, /alerts, /summary, /config,
  /resources) return 404 — the static handler has no SPA fallback.
• The checked-in static assets contain macOS duplicates:
  asset-manifest 2.json, dataflint-favicon 2.ico, dataflint-logo 2.png.

Happy to open separate issues for those.

🤖 Generated with Claude Code